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ABSTRACT 

The goal of this study is to show how some concepts 
and techniques from disciplines outside Instructional Systems 
Development (ISD) have the potential to extend and enhance the 
traditional view of ISD practice when they are employed very early in 
the ISD process. The concepts and techniques employed were 
user-centered in design and usability, and the context was an 
instructional development project, Indiana University Center for 
Excellence in Education's (CEE) Virtual Textbook, in the earliest 
stages of design. One component of the Virtual Textbook is an 
interactive whiteboard which students will use during groupwork 
activities. The major goal for the interactive whiteboard study was 
to understand how representative virtual textbook users, high school 
students, would interact with the whiteboard with the fewest possible 
imposed restrictions of an established interface. Five groups of 
students were videotaped as they constructed a timeline depicting the 
history of rock and roll music. Findings from sessions and results 
from discussions with designers are discussed. Each student group 
employed common strategies and tactics to manipulate either a picture 
or text it em on the simulated shared space. Each group also expressed 
similar social characteristics in order to manage and complete the 
timeline. Common strategies and tactics include types of moves; 
command language; students' language; shared space maintenance; item 
maintenance; and shared space navigation. Common social aspects 
include common ownership; turn-taking; strategy discussions; limited 
collaborative skills; and need to consult at close range. Designers 
had four major types of reactions to the findings and viewing of 
videotaped sessions: (1) proposing new and revised features; (2) 
reconsidering assumptions about students using the whiteboard; (3) 
questioning the design of the usability sessions; and (4) posing 
questions to be answered in future usability sessions. (Contains 11 
references.) (MAS) 
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The overall goal of this study is to show how some concepts and techniques from disciplines outside 
Instructional Systems Development (ISD) have the potential to extend and enhance the traditional view of 
ISD practice when they are employed very early in the ISD process. The concepts and techniques we 
employed were user-centered design and usability, and the context of our study was an instructional 
development project in the earliest stages of design. 

What are user-centered design and usability? 

We will define user-centered design as a philosophy that places the eventual user of a product at the 
heart of the design process for that product. Usability is a key concept within the philosophy, and refers to 
the cluster of attributes commonly considered to determine the ease-of-use a product will enjoy; learnability, 
efficiency, memorability, error prevention and recovery, and user satisfaction (Nielsen, 1993). The terms 
are sometimes used interchangeably, and both are used frequently to distinguish product development 
methods and processes that facilitate the creation of usable computer products (Strong, 1995). The methods 
and processes associated with user-centered design are derived from multiple disciplines including, among 
others, human factors, ergonomics, psychology, ethnographic studies, and document design. 

One of the fundamental principles of user-centered design is that users must be involved in uesign 
decisions for the products they will use. From this principle is derived an iterative approach to dciqn, 
since the input of users forces designers to reconsider their own decisions and assumptions during 
development process. Furthermore, the iterative approach demands use of rapid prototyping techniques to 
make testable products available early enough in the process that they may be altered, sometimes repeatedly, 
before they are delivered in final form. 

Applying user-centered design and usability to instructional product development 

A variety of principles and methods associated with user-centered design appear to overlap in part with 
elements of the instructional development process, specifically needs analysis, audience or learner analysis, 
and formative evaluation (Dick & Carey, 1990), but there are not direct equivalents for usability evaluations 
within standard ISD. Tripp and Bichelemeyer (1989) propose that rapid prototyping and testing techniques 
from user-centered design can be best applied in: 1) novel designs: those products that are new and/or; 2) 
problematic designs: those designs that present problems to the development team that have not been 
successfully solved previously. We chose the Indiana University Center for Excellence in Education's 
(CEE) Virtual Textbook project as a viable candidate for application of usability evaluation methodology to 
the development of an instructional product. 

Applying user-centered design to the concept of an interactive whiteboard 

The Virtual Textbook project, still in early design stages is defined by its developers as a "single 
integrated delivery environment" for "a new form of interactive learning'* (Siegel, 1992). One component of 
the virtual textbook is the interactive whiteboard which the designers envision that students will use during 
group work activities. As it is conceptualized, students would transmit text and pictures, among other 
things, through their virtual textbooks onto the whiteboard and share these items with other members of 
their group. 

There have been some references to interactive whiteboard technology in corporate settings (e.g., 
Weiser, 1991), but we have found no published studies on incorporating an interactive whiteboard in 
educational settings; hence, the interactive whiteboard is a novel design problem for this team. 
Consideration of the interactive whiteboard raises specific issues for the design team, for example, once an 
item is on the whiteboard, will it be owned by the group or by the original student? The team also faces a 
larger question: how should the interface between the virtual lextbook and the interactive whiteboard be 
designed? We decided to focus on the issues surrounding the interactive whiteboard for our usability study. 



3 

564 



Our major goal for the interactive whiteboard study was to understand how 
representative virtual textbook users, high school students, would interact with ar interactive whiteboard 
with the fewest possible imposed restrictions of an established interface. We chose to simulate the 
interactive whiteboard, since use of the commercially-available model would require us to conform to an 
established interface. To simulate the interactivity of a whiteboard, we used volunteers who put up and took 
down pictures and text from a regular blackboard in response to students* requests. Students participating in 
the study were told to direct their requests to the whiteboard instead of the volunteer, and given only 
minimal instruction on formulating those requests. 

We videotaped five groups of either three or four students as they constructed a timeline depicting the 
history of rock and roll music, an activity that teachers told us was representative of groupwork projects, 
but which did not overlap with any current projects for these students. Each student was given twenty 
items, both pictures of rock and roll musicians and text descriptions of these musicians, and the group was 
given guidelines for the construction of their timeline which they completed during two half-hour sessions. 
Since the goal of the usability sessions concentrated on the interactions of students with the simulated 
technology we did not focus on the timeline produced by each group, but on their interactions with the 
simulated whiteboard. Sugar and Boling (in press) give more detail on the sessions themselves. 

Results of usability sessions 

There were two major results from these early usability sessions: findings from the sessions and results 
from discussions with designers. The findings from the sessions are grouped into strategies/tactics and 
social issues. We presented the findings to the Virtual Textbook designers along with several of the 
usability session videotapes and recorded their reactions. The designer's reactions fell into four categories, 1) 
proposing new and revised features, 2) reconsidering assumptions about students using the whiteboard, 3) 
questioning the design of the usability sessions, and 4) posing questions to be answered in future usability 
sessions 

Findings from the usability sessions 

After viewing each videotaped session, we identified two major categories of student collaborative 
interactions, namely: 1) strategies/tactics; 2) social aspects. Each student group employed common 
strategies and tactics during both sessions. Students used these strategies and tactics to manipulate either a 
picture or text item on the simulated shared space. In addition to these manipulations, each group expressed 
similar social characteristics in order to manage and complete the rock and roll music timeline. Below, we 
have described common occurrences in both these categories. 



Strategies/tactic?; 

• Types of moves: There were three primary moves that the student groups performed: singular 
moves, combination moves (e.g., "Put The Beatles at 1969 and take Jerry Lee Lewis down"), and 
simultaneous moves (those moves that were voiced by more than one subject). 

• Command language: Each student group use a similar command language pattern in 
communicating to the shared space. In the beginning of the task, students constructed their 
commands with a "verb/subject" format (e.g., "Put Chuck Berry in 1950"). Students gradually 
deleted the verb from the command format and referred to the material by the subject (e.g., "Chuck 
Berry in 1950"). When student groups were involved with task and not cognizant of 
communicating formally with the shared space, they relied on using jargon to communicate with 
the shared space. Examples of these type of jargon included such statements as: "Slap on The 
Byrds close to 1963" or "Call up Prince". 

• Students' language: During group discussions, the volunteer simulating the interactive 
whiteboard could not always tell whether students wanted to put an item on the board. One student 
would say that they should put an item up, but another student would disagree, and the language 
they employed might or might not be construed as direct commands to the interface. 
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• Shared space maintenance: Since the student groups only had a limited amount of space (a 
maximum of 21 items), the groups eventually confronted the issuing of "cleaning up" or 
maintaining the shared space. The two groups employed two major strategies in dealing with this 
issue. Some student groups issued global commands; for example, one group asked to take all of 
the text items off the shared space (e.g., "Take all of the written statements down"). Another 
major strategy was to designate a certain portion of the shared space to review items. 

• Item maintenance: Each student group employed common tactics in maintaining their work 
space. There was an apparent need to sort through their items on their desks. Students would sift 
their items to figure out which items to include on the timeline; they would put used items in a 
separate pile. This finding and the previous one indicate a need for a workspace to view several 
items. This proposed workspace could be located on student's individual virtual textbook or on the 
common shared space. 

• Shared space navigation: Before students would put up a item, they checked the whiteboard. 
There was a constant interplay of students checking the board, their respective piles of pictures and 
text, checking the board again, and then deciding which item needed to be on the board. Students 
also needed to figure out a specific location to put their items on the shared space. Students either 
referred to a specific year on the timeline or referred to a existing item on the whiteboard near 
which to place pictures or text. 



Social aspects 

• Common ownership: A major question prior to this study was whether students had common 
ownership of the materials that were held in shared space. That is, if one student put up his or her 
picture in the shared space, would another student freely move or take down that student's picture? 
Overwhelmingly, each sr^ut group treated the shared space material as common property. 
Students moved other student'' materials to other locations in the shared space and deleted those 
materials off the board. When there was a disagreement on the transfer or removal of a certain 
item, it was settled by the dominant group members in that student group, not the original 
"owner" of the picture or text. 

• Turn-taking: We expected to see students establish a set of rules on how to take turns in 
communicating with the shared space. It appears that there were usually one to two dominant 
students who acted as "gatekeepers" or spokespersons for each student group. These students 
would direct other students moves. They would either approve or veto the group's decisions. This 
transfer of power originated from students' need to confirm decisions among group members before 
putting items on the board. These decisions were mediated through these dominant student(s). 

• Strategy discussions: After the initial period of putting items on the board, students usually 
discussed on strategies on how to complete the timeline. These discussions focused on particular 
pictures (e.g., "who has Jimi Hendrix?") and where to place items (e.g., "where should we put Bob 
Dylan?"). 

• Limited collaborative skills? The student groups understood their task to be a cooperative one, 
since they were told that they each had part of the materials required to complete the timeline, and 
no one of thein had everything required. However, none of the groups was observed to organize 
themselves in preparation for the work, choose a leader (although dominant members emerged in 
each group), settled on or even discuss a process for their work, or carry out any coherent effort to 
plan the task. Various groups were observed to spend time ineffectively, marginalize one or more 
members who were later discovered to have information needed by the group, and backtrack 
through actions due to misunderstandings. 

• Need to consult at close range?: Even though there was a restriction on sharing items among 
themselves (in order to maximize observed interactions with the whiteboard, and to simulate the 
probability that virtual textbook screens might not be large enough to use for this kind of 
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consultation) and groups were reminded of the restriction, students consistently consulted with each 
other by holding up and passing around pictures and text items. 



Results from discussions with Virtual Textbook designers 

The virtual textbook designers had four major types of reactions to our presentation of these 
findings and their viewing of videotaped sessions; 1) proposing new and revised features, 2) reconsidering 
assumptions about students u&ing the whiteboard, 3) questioning the design of the usability sessions, and 4) 
posing questions to be answered in future usability sessions. We believe these reactions gives insight into 
the potential impact of the usability sessions on the process of designing instructional products and the 
possible impact of user-centered design on the instructional systems design (ISD) process. 

• Proposing new and revised features for the whiteboard: The designers had envisioned the 
functionality of the interactive whiteboard within the virtual textbook environment to the extent of 
representing some of its major features in a "conceptual video" mock-up, but once they were 
presented with findings of the usability sessions and the videotapes, they began to speculate on 
both new and dramatically revised features the whiteboard might have. Among other things, they 
discussed the development of a new command set within whiteboard interface, and designing a 
horizontal whiteboard instead of a vertical one. 



• Reconsidering assumptions about students using the whiteboard: The designers showed 
considerable interest in the findings concerning strategies/tactics and social aspects described in the 
preceding section and spent time discussing how to redesign and reconceptualize the interactive 
whiteboard in response to \h.s more detailed view of the students who would use the technology. 
Should the whiteboard interface try to compensate for limited collaborative skills? If so, how? 
Should the whiteboard interface encourage or discourage the "gatekeeper" function observed in 
every group? 

• Questioning sessions' design: The designers wondered whether students were comfortable with 
addressing the "simulated" whiteboard and suggested that future usability sessions should focus on 
students groups working without an interactive whiteboard to answer some of their new questions 
about how students work together and to limit the admittedly confounding effect of the simulated 
technology. The designers were enthusiastic about the possibility of conducting further usability 
sessions, in spite of their questions about the design of this one. 

• Posing questions to be answered in future usability sessions: The designers posed a variety of 
questions they wanted answered through future studies: do students need to touch their documents 
rather than indirectly manipulating them? how should students cue their commands with the 
interface? what type of order needs to be imposed with a common work space? will the same 
social dynamics occur among students if the usability sessions were altered? what impact is the 
whiteboard technology having with the social dynamics? Though there were no conclusions 
among designers about the design of the Virtual Textbook interactive whiteboard interface, there 
was tremendous amount of creativity exhibited by the design team during the discussions of these 
usability sessions. 



567 



o 



Early usability testing model 

The model represented in this paper describes how we conducted usability tests in relation to the design 
and development work ongoing in the Virtual Textbook project. Although our study used two separate 
teams to conduct tests and carry out ongoing development, the ^odel could as easily describe parallel 
activities by members of a single team. The model cycles back into itself, demonstrating the difference 
between usability studies and experimental research; the focus of usability studies is contained within a 
project and the results of the studies guide further design efforts. There is no expectation that findings from 
these studies will generalize outside the project for which they are conducted, although they may be of great 
interest to designers working on similar projects. 




Some anticipated criticism of the early usability testing model is that this testing is simply formative 
evaluation. Tessmer (1993) points that the rapid prototyping technique is similar to "repeated formative 
evaluations." Although the evaluation techniques are similar in usability testing and traditional formative 
evaluation, the key difference between the two is how early in the process and how often a product is 
evaluated. In describing the rapid prototyping techniques of his ID2 theory, Merrill and his associates 
comment that the instructional systems development (ISD) approach assumes that there will be no actual 
product until the later stages of the process (Jones, et al, 1992). The early usability testing model provides 
designers the opportunity to assess and evaluate their concepts and assumptions during the earliest stages of 
a design. The model also presumes that the evaluation process will be carried out several times during the 
development process, giving designers the opportunity to adjust designs in situations where they have little 
prior experience to gauge the probable success of a product. 

Implications for the ISD process 

What impact does this model have for the ISD process? In our experience with this study, early 
usability testing may impact the ISD process in four ways; 1) condense ISD process; 2) refine prescriptive 
methods; 3) encourage a cyclical process; and 4) concentrate attention on the human elements of the 
process. 
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• Condenses 1SD process: The results of user-centered design evaluations provide insight for the 
next analysis phase, and can give rise early in the process to questions that might not otherwise be 
asked until a product is almost ready to be implemented. Catching and considering a wider range 
of issues for design at the beginning of a project may reduce overall development time, especially 
in cases of novel or problematic technology. 

• Refines prescriptive methods; Early usability testing also gives instructional designers the 
opportunity to test the prescriptive methods they are encouraged to develop by Reigeluth (1983) 
Through each redesign phase of the early usability testing model, designers can refine these 
methods with the goal of having the most effective methods for their final product. We would 
assert that refined methods developed using an early usability testing model must be more effective 
than prescribed methods tested only once under a traditional ISD model. 

• Encourages a cyclical process: Dumas and Redish (1993) differentiate between two types of 
process models: assembly line and cyclical team models. In the assembly line process, designers 
develop their products in successive and separate phases. There is a dearth of feedback between 
each of the phases. An example of this model is where a design team develops a product during 
the development phase, and then, the marketing department (a different set of individuals) evaluates 
the same product. There is limited communicadon between these two groups and thus, 
assumptions and realizations of the design team are not communicated to the marketing 
department. In contrast, the cyclical team process, a design team performs each step of the design 
process (e.g., analysis, development, evaluation). The assumptions and realizations of the design 
are fully communicated to each member on the design team. The early usability testing model is 
explicitly cyclical, encouraging both communication between all team members and re- 
examination of prior decisions on each repetition of the cycle. We are nor suggesting that all 
instructional designers who follow the traditional ISD process are using an assembly line process. 
However, the traditional ISD model presents cyclical development as a desirable option, not as a 
requirement 

• Concentrates on the human element in the process: Early usability testing seems to give the 
eventual user of an instructional product a consistent "presence" within the development process. 
Once designers have observed real people use even an approximation of their product at the earliest 
stages of conception, they tend to refer back to those people throughout the project to visualize the 
effect that future design changes might have on them. 

Conclusion 

Our study has only touched the surface of the issues, and we know that some of the controversies that 
attend user-centered design accompany its introduction into ISD, for example, the question of cost/benefit 
analysis for the additional effort spent conducting usability sessions and debates over specific 
methodological applications for one type of project and another. However, our experience suggests that the 
possibilities for incorporating user-centered design concepts, and specifically usability testing, in the ISD 
process deserve consideration and further study. 
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